Associating LIDAR data and image data

ABSTRACT

A monocular image often does not contain enough information to determine, with certainty, the depth of an object in a scene reflected in the image. Combining image data and LIDAR data may enable determining a depth estimate of the object relative to the camera. Specifically, LIDAR points corresponding to a region of interest (“ROI”) in the image that corresponds to the object may be combined with the image data. These LIDAR points may be scored according to a monocular image model and/or a factor based on a distance between projections of the LIDAR points into the ROI and a center of the region of interest may improve the accuracy of the depth estimate. Using these scores as weights in a weighted median of the LIDAR points may improve the accuracy of the depth estimate, for example, by discerning between a detected object and an occluding object and/or background.

BACKGROUND

Camera images traditionally include two-dimensions of data. Therefore,even when object detection is conducted on an image of a scene, thisdetection provides no more than the coordinates of the image thatcorrespond to the detected object (i.e. depth and/or scale isambiguous). Solutions, such as using stereo cameras, have beenintroduced to recover the depth of a detected object from an image.However, stereo camera depth detection is error-prone and is often tooslow for real-time applications, such as autonomous vehicle control,which could lead to reduced safety outcomes.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentify the figure in which the reference number first appears. Thesame reference numbers in different figures indicate similar oridentical items.

FIG. 1A illustrates an example image that includes a detected object andbounding box generated to identify the location of the detected objectin the image.

FIG. 1B illustrates a block diagram of an example scenario illustratingthe depth perception problem.

FIGS. 2A & 2B illustrate a birds-eye perspective and a side profile,respectively, of an example scenario, and LIDAR data that may correspondto the elements of FIG. 2C.

FIG. 2C illustrates an example image that includes an example detectedobject, an example region of interest, an example occluding object, andexample LIDAR data projected onto the image.

FIG. 3 illustrates a block diagram of an example probabilitydistribution generated by a machine-learned model that corresponds tothe example scenario of FIGS. 2A-2C, and example probabilitiesassociated with three example LIDAR points.

FIG. 4A illustrates a side profile view of depth measurements associatedwith a few select example LIDAR points for discussion in subsequentfigures.

FIG. 4B illustrates projections of the select example LIDAR points intothe region of interest for discussion in subsequent figures.

FIG. 4C illustrates an example distribution for generating a factor forLIDAR points based at least in part on distance of the lidar pointprojections from a center of the region of interest.

FIGS. 5A-5C illustrate an example process for determining a depthestimate for an objected detected in an image.

FIG. 6 illustrates a block diagram of an example autonomous vehicle thatmay incorporate the vision-metaspin association system discussed herein.

DETAILED DESCRIPTION

The techniques (e.g., machine and/or process) discussed herein mayinclude determining a distance to an object in an environment usingimage data from an image sensor and LIDAR data from a LIDAR sensor. Insome examples, the techniques discussed herein determine the distancefrom the camera to the object (e.g., a depth of the object) based atleast in part on receiving an indication of pixels that correspond tothe object in the image (termed a “region of interest” herein),receiving LIDAR data; and determining LIDAR points from the LIDAR datathat correspond to the region of interest and a time the image wastaken. Once these LIDAR points have been identified, the techniques mayinclude scoring the LIDAR points and using the scores associated withthe LIDAR points as weights in a weighted median calculation, sortingthe LIDAR points by distance (e.g., each LIDAR point may be associatedwith a distance measurement and, in some examples, an angle as well),and determining a weighted median of the sorted LIDAR points using thescores as the weights. In some examples, the techniques may includeidentifying the weighted median as a depth estimate to associate withthe object. Such techniques may provide for more accurate depthestimates of objects by accounting for LIDAR data of occluding objects.

The images discussed herein may be monocular images capturing atwo-dimensional representation of an environment. That is, the monocularimages may include color/greyscale image data (including, but notlimited to, visible camera data, infrared camera data, and the like),but lack a depth (e.g., a “z-axis” in a Euclidian coordinate system).The techniques discussed herein may include determining a depth of anobject detected in the image. In other words, the techniques discussedherein identify how far away from where the image was taken (e.g., thecamera, the focal plane, the image plane—although the discussion hereinsimplifies this by just referring to the “camera,” even though the imageplane may be at a slightly different position than the camera because oflens characteristics) the detected object lies. In some examples, aLIDAR sensor may measure the distance from the LIDAR sensor to multiplesurface points within a scene. For each surface point, the LIDAR sensormay determine both the distance of the surface point and its angulardirection with respect to the LIDAR sensor. This capability may be usedto create a point cloud including three-dimensional coordinates of themultiple surface points. In some examples, LIDAR sensors are configuredto rotate in 360 degrees to create a point cloud (e.g., a plurality ofLIDAR points) of an environment surrounding the LIDAR device that iswithin a field-of-view (“FOV”) of the LIDAR sensor, though any othertype of LIDAR sensor is contemplated (e.g. solid-state, MEMS, flash,etc.). When multiple LIDAR devices are used simultaneously, all LIDARdata collected over a period of time (such as a single spin for spinningLIDAR devices) is referred to herein as a “metaspin.”

In some examples, the techniques may include capturing an image of anenvironment by a camera and creating a point cloud of the environmentusing a LIDAR sensor. The techniques may include detecting an object inthe image and/or determining a region of interest (ROI) associated withthe detected object (e.g., a mask comprised of pixels corresponding tothe detected object, a bounding box that encompasses pixels identifiedas being associated with the detected object, and the like). Forexample, the ROI may correspond with a visible surface of the detectedobject, although the distance from the camera to the visible surface ofthe detected object (the “depth”) may be unknown, if only a monocularimage is available.

In some examples, the techniques may include identifying a portion ofLIDAR data that corresponds to the portion of the environment capturedin the image by the camera and/or that corresponds to the ROI of theimage, which may be a smaller subset of the image. The techniques mayadditionally or alternatively include determining LIDAR data that mostclosely corresponds to a time at which the image was captured. In someexamples, the camera and the LIDAR sensor may be phase-locked so thatthe camera and the LIDAR sensor capture data corresponding to a sameregion of the environment at the same time, but, in some examples, thecamera and the LIDAR sensor may capture data corresponding to the sameregion at slightly different times. In the latter example, thetechniques may include determining LIDAR data that was captured at atime that most closely corresponds to a time that the image wascaptured. For example, if a camera captures images of the region of theenvironment at 30 Hz and the LIDAR sensor captures LIDAR data of theregion at 10 Hz, the techniques may include determining, for every threemetaspins of the LIDAR sensor, which metaspin of the three includes asubset of data that corresponds most closely to the image in time (andcorresponds to the ROI, as mentioned above). Similarly, in someexamples, multiple images may be chosen representative of the time ametaspin was collected and a subset of the images may be selected as theimages most representative of the environment as the metaspin wascollected.

Unless discussed otherwise, the term “LIDAR points” refers to the subsetof the LIDAR data taken in a metaspin that corresponds most closely tothe ROI in space (where the ROI corresponds to in the environment and/orimage) and/or time.

In some examples, once the LIDAR points have been identified thatcorrespond to the ROI and/or the time, as discussed above, thetechniques may further include scoring these LIDAR points, sorting theLIDAR points by distance (e.g., each LIDAR point is associated with adepth measurement that includes at least a distance from the LIDARsensor and an angle and the sorting may include organizing these fromsmallest distance to greatest distance or vice versa), and identifying aLIDAR point associated with a weighted median of the sorted LIDARpoints. In some examples, the scores of the LIDAR points may be used asthe weights to find the weighted median. In some examples, thetechniques may include identifying, as a primary depth estimate, a depthmeasurement associated with the LIDAR point that is the weighted median.

However, in some scenarios, a second object may occlude at least part ofthe detected object in the image. In some cases, the second object maybe positioned such that the primary depth estimate corresponds, inreality, to the second object, which may be an occluding object if itappears in front of at least a portion of the detected object. Toaccount for this, the techniques may include removing a group of theLIDAR points that correspond to distances within a range of the primarydepth estimate. For example, any LIDAR points associated with depthmeasurements that lie between 0.8 meters in front of the primary depthestimate (i.e., towards the LIDAR sensor) and 1.6 meters behind theprimary depth estimate (i.e., on the other side of the primary depthestimate from the LIDAR sensor) may be excluded. The techniques mayinclude identifying the subset of LIDAR points associated with depthmeasurements that lie outside of this range, sorting the subset of LIDARpoints, identifying a weighted median of the subset of LIDAR points, andidentifying the weighted median of the subset as a secondary depthestimate.

To distinguish between the primary depth estimate and the secondarydepth estimate as truly being associated with the detected object andnot an occluding object, the techniques may include determining adifference between the primary depth estimate and the secondary depthestimate such as, for example, a distance between the two estimates. Thetechniques may compare this to a threshold difference that may bestatically defined (e.g., 1.5 meters, 3 meters) or that may beassociated with a classification of the detected object (e.g., 6 metersfor semi-trucks, 3 meters for pickup trucks, 2 meters for passengervehicles, 1 meter for compact vehicles).

If the difference is less than or equal to the threshold difference(e.g., the difference between the two estimates is 1 meter and thedetected object is a passenger vehicle associated with a thresholddifference of 2 meters), the techniques may identify the estimates asboth corresponding to the detected object. In some examples, thetechniques may output the primary depth estimate as the final estimateand/or may average the estimates, etc.

If the difference meets and/or exceeds the threshold difference (e.g.,the difference between the two estimates is 3 meters and the detectedobject is a passenger vehicle associated with a threshold difference of2 meters), the techniques may choose one of the primary depth estimateor the secondary depth estimate by: comparing the first depth estimateand the second depth estimate to an output of a monocular image model(e.g., a machine-learned model that takes an estimated height of thedetected object and/or a classification of the detected object as inputand outputs a probability distribution of depths that, for a specificdepth measurement, identifies a probability density that the depthmeasurement corresponds to the object); comparing a first density ofLIDAR points associated with the first depth estimate to a seconddensity of LIDAR points associated with the second depth (e.g., toidentify which is associated with a greater density and/or greaternumber of LIDAR points); and/or comparing the first depth estimate andthe second depth estimate to an object track associated with the object.In some examples, an object track may include previous positions of adetected object, a velocity of the detected object, and/or a predictedposition and/or velocity of the detected object. In some examples, oneof the primary depth estimate or the secondary depth estimate may beidentified as the output depth estimate to be associated with thedetected object. In some examples, the other of the two may be discardedor associated with an occluding object.

In some examples, scoring a LIDAR point may include determining, from aprobability distribution generated by a monocular image model (e.g., amachine-learned model that takes a detected object and/or aclassification of the detected object of an image as input and outputs aprobability distribution over representative depths), a probabilitydensity associated with the distance measurement identified by the LIDARpoint (e.g., a probability density representative of a probability perunit length). The scoring may additionally or alternatively includeprojecting the LIDAR point from three-dimensional space into the ROI intwo-dimensional space so that the projected LIDAR point is associatedwith a two-dimensional coordinate; determining a distance of thetwo-dimensional coordinate to a center of the ROI; and generating afactor (e.g., a scalar) that is based at least in part on the distance(e.g., as the distance increases, the factor decreases). In someexamples, generating the score for the LIDAR point includes multiplyingthe probability density by the factor.

In some examples, the techniques may include fusing vision data andLIDAR data into a single dataset by projecting the three-dimensionalLIDAR points into the ROI so that an individual projected LIDAR point(i.e., a “projection” of the LIDAR point into image space that istwo-dimensional) corresponds with a coordinate of an image. In someexamples, this fusion may be improved by tracking a velocity ofdeviations of the camera and/or LIDAR sensor from a normal plane (e.g.,shaking due to environmental disturbances).

The techniques discussed herein may improve functioning of a computer byequipping the computer to be able to determine the depth of a detectedobject in an image. Moreover, the techniques may improve the accuracy ofdepth estimates of an object over stereo camera techniques and/or usinga monocular image model exclusively. The techniques also reduce thenumber of image sensors necessary to provide depth perception for aparticular FOV, for example, by providing depth estimates using amonocular image as opposed to requiring multi-view or stereo geometryreconstruction. This elimination of redundant sensor(s) correspondinglyreduces the number of computational cycles necessary to achieve depthperception and reduces the consumption of other such as power and/ornetworking bandwidth. Furthermore, during preliminary experimentationthe techniques discussed herein provided a depth estimate for a detectedobject in approximately 6 milliseconds or less, making the depthestimate useful for real-time applications, such as controlling anautonomous vehicle.

Example Scenario

FIG. 1A illustrates an example image 100 that includes a detected object102, in this example a van, and ROI 104 generated to identify thelocation of the detected object in the image. The ROI 104 in FIG. 1A isindicated by a two-dimensional bounding box. However, it is understoodthat any other suitable method may be used to indicate a group of pixelsof the image that correspond to the image (e.g., a pixel maskidentifying discrete pixels associated with the vehicle, which maygenerally be referred to as an instance). In some examples, the imageand/or bounding box may be generated by a vision system of an autonomousvehicle and received by a perception system of an autonomous vehicle forthe perception system to determine a depth associated with the detectedobject.

FIG. 1B illustrates a block diagram of an example scenario 106 thatillustrates the depth perception problem (or scale ambiguity) morethoroughly. FIG. 1B depicts an example vehicle 108 (e.g., an autonomousvehicle that includes a camera) that has taken an image and detected anobject (e.g., a vehicle 110) within the image. The example vehicle 108may have identified pixels corresponding to the detected object 110using a bounding box, however, the image only provides positional datain two dimensions, horizontally and vertically relative to the camera'sposition. Therefore, the image is insufficient to identify the depth ofthe detected object 110 relative to the camera—the detected object 110could equally be located at depth 112 or depth 114, which correspond tothe surface of the detected object 110 captured in the image.Hypothetically, the rays indicated by 116 originating at the camera ofthe example vehicle 108 may be bounded by the edges of the ROI, butcould extend infinitely away from the camera.

Example LIDAR Data

FIGS. 2A & 2B illustrate a birds-eye perspective and a side profileview, respectively, of an example scenario 200, example LIDAR datarepresented by stars and captured by a LIDAR sensor of an examplevehicle 202, an example detected object 204, and an example occludingobject 206 (e.g., a pole of a signpost). For example, the LIDAR datadepicted may represent the LIDAR data captured in one metaspin. It isunderstood that, in practice, a point cloud would likely contain tens ofthousands, or more, points, instead of the tens that are depicted here.The vehicle 202 may represent an autonomous vehicle equipped with atleast a camera and a LIDAR sensor.

In the depicted example scenario 200, the vehicle 202 may have alreadycaptured an image (208) using the camera, detected the object 204 in theimage 208, generated the ROI 210 to identify the location of thedetected object 204 within the image, and determined data associatedwith a metaspin that most closely corresponds in time to a time theimage was taken. The rays 212 represent the boundaries of the ROI 210,which may correspond to any point within the two-dimensional bounds ofthe ROI 210 and is therefore unlimited in a third dimension (i.e., depthin this case). Therefore, the rays (or lines) 212 are associated with afrustum corresponding to the camera (e.g., a sensor plane, an imageplane, and the like) and may continue infinitely, although a perceptionengine may limit the extent of the rays 212 by a tolerable limit of theLIDAR sensor (e.g., 150 meters) to identify the LIDAR points that mayreasonably correspond to a detected object. In some examples, RADARpoints may be used beyond the tolerable limit of the LIDAR sensor and/orRADAR data may be additionally or alternatively be used. In someexamples, where both LIDAR data and RADAR data are used, the RADAR datamay be weighted more heavily at further distances (e.g., outside 150meters or 100 meters of the vehicle 202) and LIDAR more heavily atcloser distances (e.g., within 150 meters or 100 meters of the vehicle202). It is also contemplated that LIDAR data may be heavily weighted atfurther distances and RADAR data at closer distances. Although thespecification discusses LIDAR data, the techniques discussed herein mayequally be applied to a system that receives data from any sensor thatdetects a three-dimensional position of surface(s) (e.g., LIDAR, RADAR,any sensor that may generate a point cloud or other representation ofsurfaces of an environment).

In some examples, a perception system may determine which LIDAR pointscorrespond with the ROI 210 based at least in part on a position and/ororientation of the camera in space relative to a position and/ororientation of the LIDAR sensor, distances and angles associated withindividual points of the LIDAR data, and/or the rays 212. The LIDARpoints determined to correspond to the ROI 210 are shown by shadedstars, such as LIDAR point 214, whereas the remaining LIDAR points thatlie outside the ROI 210 are shown with white centers, such as LIDARpoint 218.

The depicted example includes LIDAR points that correspond with the ROI210 and that include a cluster 220 of LIDAR points that correspond to asurface of the detected object 204, a cluster 222 of LIDAR points thatcorrespond to a surface of the occluding object 206, and a LIDAR point214 that corresponds to a surface of an object in the background of theimage 208.

In some examples, once the perception engine identifies the LIDAR pointsthat correspond with the ROI 210 (indicated by the shaded stars), theperception engine may project the LIDAR points (i.e., clusters 220 and222 and point 214 in this example) into the image 208, as depicted inFIG. 2C. This may include projecting LIDAR points to corresponding imagecoordinates, as would be appreciated. Additionally or alternatively,this may include projecting three-dimensional LIDAR points intotwo-dimensional projected LIDAR points (i.e., projections). Note thatthe number of LIDAR points in the cluster 222 corresponding to thesurface of the detected object 204 have been reduced to two points inFIG. 2C for simplicity.

Example LIDAR Point Scoring

FIG. 3 illustrates a diagram of an example probability distribution 300generated by a monocular image model that corresponds to the examplescenario of FIGS. 2A-2C and example probabilities associated with threeexample LIDAR points, 302, 304, and 306.

In some examples, to identify the depth of the detected object, amonocular height may be used that takes as input an objectclassification and/or the ROI 210. U.S. application Ser. No. 15/453,569,titled “Object Height Estimation from Monocular Images” and filed Mar.8, 2017 describes such a model and is incorporated herein by reference.The monocular image model may include a machine-learned model such as,for example, a convolutional neural network (CNN). In some examples, themonocular image model may receive an image (e.g., the ROI 210) and/orobject classification as input and may output a probability distributionsimilar to the example probability distribution 300.

In some examples, as in FIG. 3, the probability distribution 300 mayinclude a series of bins, each bin representing an estimated size rangefor the object and/or an estimated distance of the object. FIG. 3depicts the latter case—different bins are associated with a range ofestimated distances and a probability. For example, the probability maybe a probability that the distance measurement is associated with theobject, based on a classification of the object and/or a height estimateof the object. As a non-limiting example, an output having eight binsmay represent depth distributions according to 0-2 m, 2-4 m, 4-6 m, 6-8m, 8-10 m, 10-100 m, where the value associated with each bin isindicative of a probability that the depth associated with the data lieswithin that bin. Although the bins are depicted in FIG. 3 as being ofequal width, it is understood that the bins may have different widths(e.g., the bin widths may be calculated to correspond to a quarter or ahalf of a standard from a mean of the probability distribution). In someexamples, the first and last bins may represent data not under a minimumdepth and not over a maximum depth. In some examples, distributions inthe first and last bins may be scaled (e.g., linearly, exponentially, asa Gaussian, or any other distribution). In examples where the systemestimates the depth of the detected object 204 using only an output ofthe monocular image model, the system may retrieve an estimated size ofthe object from a bin associated with the highest probability.

In the depicted example, and in a system using the improved techniquesdiscussed herein, the techniques may include identifying, from theprobability distribution 300, a probability that corresponds to a depthmeasurement associated with a LIDAR point. For example, in FIG. 3, LIDARpoint 306 is associated with a lowest probability, LIDAR point 302 isassociated with a slightly higher probability, and LIDAR point 304 isassociated with a highest probability of the three LIDAR pointsdepicted.

In some examples, the techniques may include determining, from theprobability distribution 300, a probability density that corresponds toa depth measurement identified by a LIDAR point. A probability densitymay be the height of a bin (i.e., the probability) modified (e.g.,divided) by a width of the bin, although other methods of calculatingthe probability density to account for the height and width arecontemplated. This probability density may be indicative of aprobability density associated with the distance measurement.

In some examples, the probability distribution 300 may further include amean, standard deviation, and/or a confidence score. The techniquesdiscussed herein may include generating a probability distribution overestimated depths for an ROI, such as ROI 210, identifying a probabilityand/or probability density associated with each LIDAR point. In someexamples, the techniques discussed herein include inputting the ROI intothe monocular image model and, in some examples, may additionally inputan object classification into the monocular image model as well (e.g.,an indication determined by a perception system of the vehicle 202 suchas, for example, “passenger vehicle,” “compact vehicle,” “deliverytruck,” “semi-truck,” “pickup truck,” “bicycle,” “pedestrian”).

FIG. 4A illustrates a side profile view of depth measurements associatedwith a few select example LIDAR points for discussion in subsequentfigures. These example LIDAR points include a cluster of LIDAR points400 associated with a surface of the occluding object 206, two points(402 & 404) associated with a surface of the detected object 204, and aLIDAR point 406 associated with a surface of an object in a backgroundof the ROI 210.

FIG. 4B illustrates example projections of the example LIDAR points400-406 into the corresponding image for discussion in subsequentfigures. In some examples, a perception engine of the vehicle 202 mayproject three-dimensional LIDAR points 400 into the image (which shouldproject into the ROI 210) to produce two-dimensional LIDAR projections.Projected LIDAR points 400′ may be the projections of LIDAR points 400into image space. Projected LIDAR point 402′ may be a projection ofLIDAR point 404 into image space. Projected LIDAR point 404′ may be aprojection of LIDAR point 404 into image space. Projected LIDAR point406′ may be a projection of LIDAR point 406 into image space. In someexamples, projecting an individual LIDAR point into the image mayassociate the individual LIDAR point with an image coordinate based atleast in part on the coordinate being a nearest coordinate to theprojection of the individual LIDAR point into the image.

The techniques discussed herein may include generating scores for thoseLIDAR points determined to correspond to the ROI 210 in spatial positionand time. The perception engine of the vehicle 102 may generate a scorefor an individual LIDAR point based at least in part on a factor thatmay be inversely proportional to the distance of the projectedindividual LIDAR point from a center of the ROI 210, as FIG. 4Cillustrates. In some examples, this distance may be fit to or otherwisecorrespond with a factor defined by a two-dimensional Gaussiandistribution normalized to the size of the ROI 210 and/or a parabola togenerate the factor, though any other association is contemplated (e.g.,Euclidian distance, linear, quadratic, polynomial, etc.). In someexamples, the distribution may be normalized such that a furthest edgeor corner of the ROI 210 is two standard deviations from the center ofthe ROI 210.

FIG. 4C depicts an example of a distribution 408 that includes contourrings 410, 412, 414, and 416, which indicate decreasing values of thefactor based on increasing distance from the center 418 of the ROI. Forthe sake of discussion, center 418 may corresponding to a point (x=0,y=0) in Euclidian space, although the vision system may reference pixelsof the ROI in any other suitable manner. FIG. 4C also depicts projectedLIDAR points 400′-406′ without including their number identifiers, forclarity. FIG. 4C also shows a representation 420 of the distance scorefunction, such as may be determined passing through the line 422, y=0,and the factor values that correspond to the contour rings 410, 412,414, and 416. For example, the center 418 is associated with a greatestfactor value 424. The factor value at a point in the ROI 210corresponding to y=0 and an x-value defined by contour ring 410 may befound to be a factor value 426 so on for y=0 and x-values defined bycontour rings 412 and 414, which may correspond to factor values 428 and430, respectively. Though illustrated in FIG. 4C pictorially, such afactor may be determined using the following equation:

${C = {Ae}^{- {(\frac{{({x - x_{c}})}^{2} + {({y - y_{c}})}^{2}}{d^{2}})}}},$

where A represents some defined maximum score, (x_(c), y_(c)) representsa center of the ROI 210 in image coordinates, and d represents somedesired factor associated with the width of such a distribution.

It is understood that, although the score or factor is depicted as aGaussian distribution, any suitable distribution could be used such as,for example, a scalar based purely on Euclidian distance, a distributionthat includes multiple local maxima (e.g., where multiple objects aredetected or for particular types of environments, such as using Gaussianmixture models), a parabola, etc, as well as any inverse of theaforementioned scoring functions (e.g. scoring functions which increaseas the point moves away from a center of an ROI).

In some examples, the techniques discussed herein may includedetermining a factor (e.g., a scalar) for the projected LIDAR pointsbased at least in part on a distance of the (two-dimensional) projectedLIDAR points from a center 418 of the ROI 210. This factor mayadditionally or alternatively be based on a distribution determinedrelative to a distance from the center 418, as discussed above.

In some examples, the techniques discussed herein may generate anoverall score for the LIDAR points corresponding to the ROI 210 in spaceand time, where generating an overall score for an individual LIDARpoint may be based at least in part on a probability and/or probabilitydensity associated with a probability distribution generated by themonocular image model, as discussed in connection with FIG. 3, and/or afactor associated with the individual LIDAR point, as discussed inconnection with FIG. 4C. In some examples, the score may be generated bymultiplying the probability and/or probability density by the factor.

Example Process

FIGS. 5A-5C illustrate an example process 500 for depth perception for amonocular image (e.g., determining a depth estimate for an objecteddetected in an image). In some examples, the example process 500 may beconducted by a vision engine 502 and/or a perception engine 504. It isunderstood that, although some operations are depicted as beingconducted by one of these engines, it could additionally oralternatively be conducted by the other engine. In some examples, thevision engine 502 and/or a perception engine 504 may be part of anautonomous vehicle system for controlling an autonomous vehicle. In someexamples, the vision engine 502 and the perception engine 504 mayconduct one or more of the operations discussed herein in parallel. Forexample, FIGS. 5A and 5B illustrate the vision engine 502 and theperception engine 504 operating in parallel. It is also understood thatthe vision engine 502 and the perception engine 504 may conduct one ormore of the operations serially (e.g., where an operation at one enginerequires a product of an operation at the other engine).

At operation 506, the example process 500 may include receiving an imageof an environment, according to any of the techniques discussed herein.In some examples, the image may be a monocular image (whether color(e.g., RGB), grayscale, IR, UV, or the like), although it is understoodthat the image may be a stereo image (or otherwise multi-view image) andthat the example process 500 may also be used to improve or verifydepths associated with such an image. In some examples, a camera on anautonomous vehicle may capture the image.

At operation 508, the example process 500 may include detecting anobject (e.g., object 102, object 204) in the environment from the image,according to any of the techniques discussed herein. In some examples,the perception engine 504 may detect the object.

At operation 510, the example process 500 may include generating an ROI(e.g., ROI 104, ROI 210) corresponding to the detected object, accordingto any of the techniques discussed herein. For example, operation 514may include generating a bounding box, instance segmentation, mask, orother identifier of image coordinates (e.g., pixels) associated with theobjected detected in the image. Although depicted as two operations, itis understood that operations 508 and 510 and/or any other pair ofoperations may be performed substantially simultaneously. That is, animage may be put into a detector, an output of which is an indication(e.g., one or more bounding boxes) of detections of a particular objector objects. In some examples, example process 500 may begin by receivingthe ROI and/or object data (e.g., an object classification).

At operation 512, the example process 500 may include receiving LIDARdata and/or determining LIDAR points of the LIDAR data that correspondsto the ROI and/or a time at which the image was captured, according toany of the techniques discussed herein. For example, see the shadedstars of FIGS. 2A-2C, as opposed to the white-filled stars of FIGS.2A-2C. In some examples, this may additionally or alternatively includeRADAR points received from a RADAR sensor. In some examples, RADAR datamay be used for data points beyond a maximum range of the LIDAR (e.g.,100 meters). In some examples, determining the LIDAR data thatcorresponds in space and time to the ROI includes a geometriccalculation based on a known position and orientation of the camera andLIDAR sensor and depth measurements associated with the LIDAR points. Insome examples, “depth measurement” associated with a LIDAR point mayinclude a distance from the LIDAR sensor and an angle relative to anaxis of the orientation of a LIDAR emitter/receiver pair. In anadditional or alternate example, determining the LIDAR points thatcorrespond to the ROI may include projecting the LIDAR points into animage space corresponding to the ROI and determining the LIDAR pointsthat are associated with image coordinates within the ROI.

At operation 514, the example process 500 may include generating, via amonocular image model such as the model discussed in FIG. 3, aprobability distribution of depths for the detected object in the ROI ofthe image, according to any of the techniques discussed herein.

At operation 516, the example process 500 may include generating scoresfor the LIDAR points, according to any of the techniques discussedherein. In some examples, this may include generating a distinct scorefor each LIDAR point of the LIDAR points. In some examples, scores maybe generated for all the LIDAR points by a same process for generating ascore for an individual LIDAR point. Generating a score for anindividual LIDAR point may include operation 516(A), which yields aprobability and/or probability density in association with the LIDARpoint, and/or operation 516(B), which yields a factor in associationwith the LIDAR point. In some examples, generating the score may includemultiplying the probability density determined at operation 516(A), bythe factor determined at operation 516(B).

In some examples, generating the score for the LIDAR point may includeassociating the probability and/or probability density with the LIDARpoint, associating the factor with the LIDAR point, and/or associating aproduct of the probability and/or probability density and the factorwith the LIDAR point. For example, determining the score for a LIDARpoint may include determining a height and width of a bin of theprobability distribution associated with a distance defined by the LIDARpoint, determining a probability density based at least in part on theheight and width, determining a factor based at least in part on adistance of the projected LIDAR point from a center of the ROI, anddetermining the score for the LIDAR point by multiplying the probabilitydensity by the factor.

At operation 516(A), the example process 500 may include determining aprobability and/or probability density to associate with the individualLIDAR point, according to any of the techniques discussed herein. Thismay include determining a point on the probability distribution thatcorresponds with a depth measurement of the LIDAR point. In someexamples, this may include associating that LIDAR point with a bin onthe probability distribution and a probability associated therewith, aswell as (at least in some examples) modulating (e.g. dividing) the valueby the width of the associated bin to determine a probability densityassociated therewith.

At operation 516(B), the example process 500 may include determining afactor to associate with an individual LIDAR point, according to any ofthe techniques discussed herein. In some examples, operation 516(B) mayinclude determining a factor for each LIDAR point. Operation 516(B) mayinclude projecting the individual LIDAR point into an image space of theROI to determine a distance from the individual LIDAR projection to acenter of the ROI. The projected LIDAR point may thereby be associatedwith coordinates in an image space of the ROI. In some examples, thefactor assigned to a LIDAR point may decrease in magnitude as a distanceof the LIDAR point's projection from the center of the ROI increases. Insome examples, this decrease may be defined by a Gaussian distribution,a Euclidean distance, a parabola, a topology that includes multiplelocal maxima, etc. See at least FIGS. 4A-4C and the accompanyingdiscussion for more details.

Turning to FIG. 5B, at operation 518, the example process 500 mayinclude sorting the LIDAR points by distance, according to any of thetechniques discussed herein. For example, the LIDAR points areassociated with depth measurements that define at least a distance and,in some examples, an angle (or angles—e.g., azimuth and elevation). Insome examples, the LIDAR points are sorted from least distance togreatest distance, but this may be reversed. In other words, the LIDARpoints define at least a distance, therefore, the LIDAR points aresorted by according a magnitude of this distance. In at least someexamples, a percentage of the furthest and closest LIDAR points may bediscarded (e.g., the closest and furthest 5%) before proceeding.

At operation 520, the example process 500 may include determining aweighted median of the sorted LIDAR points and selecting, as a primarydepth estimate, a depth measurement associated with the weighted median,according to any of the techniques discussed herein. In some examples,the scores generated for the LIDAR points may be used as the weights forthe weighted score determination. For example, for n distance-sortedLIDAR points, x₁, x₂, . . . x_(n), the weighted median may be found bynormalizing the scores corresponding to the LIDAR points, w₁, w₂, . . .w_(n) such that

${\sum\limits_{i = 1}^{n}w_{i}} = 1$and where the weighted median may be the LIDAR point, x_(k), satisfying:

${\sum\limits_{i = 1}^{k - 1}{w\_ i}} \leq {{1/2}\mspace{14mu}{and}\mspace{14mu}{\sum\limits_{i = {k + 1}}^{n}w_{i}}} \leq {1/2}$

In some examples, the primary depth estimate may include the LIDAR pointitself that corresponds to the weighted median (e.g., a distance and anangle) or, other examples, the primary depth estimate may include thedistance and/or a projection of the LIDAR point onto a ray from thecamera through a point on the detected object such as, for example, thecenter of the ROI.

In some examples, example process 500 may exclude at least operation 516and may determine a median of the LIDAR points without generating scoresfor the LIDAR points. However, in some cases, omitting the operation 516may decrease accuracy of the depth estimate.

At operation 522, the example process 500 may include outputting theprimary depth estimate to a vehicle planner for the vehicle planner tocontrol an autonomous vehicle based at least in part on the position ofthe detected object, according to any of the techniques discussedherein. In some examples, the perception engine 504 may output the ROIand the depth estimate and this may be sufficient to identify a positionof the detected object in the environment. e.g. In some examples, theperception engine 504 may output at least a position and, in someexamples, a size and/or orientation of the detected object based atleast in part on the depth estimate and/or corresponding ROI which maybe relative to a local and/or global map and stored therewith. In someexamples, the depth estimate may be used to conduct a geometriccalculation to determine a size of the detected object.

At operation 524, the example process 500 may additionally oralternatively include removing a portion of LIDAR points from the LIDARpoints, according to any of the techniques discussed herein. The portionof LIDAR points may be associated with distance measurements that lessthan and/or greater than one or more distance thresholds. For example,the perception engine 504 may remove LIDAR points associated withdistance measurements that are less than 1 meter and/or are more than 1meter from the primary depth estimate, though such distance thresholdsneed not be symmetrical. References to “in front” may be construed toinclude points lying between the primary depth estimate and the LIDARdevice and “behind” may be construed to mean lying beyond the primarydepth estimate, away from the LIDAR device. In some examples, the rangemay include distance measurements that lie within 0.8 meters in front ofthe primary distance measurement to 1.6 meters behind the primarydistance measurement. In some examples, the range may include distancemeasurements that lie within 1.6 meters in front of the primary distancemeasurement to 1.6 meters behind the primary distance measurement. Manyvariations are contemplated and the range may vary based at least inpart on an object classification associated with the detected object.For example, the range may be defined as 0.8 meters in front to 3 metersbehind for a detected object classified as a “delivery truck,” 0.5meters in front to 1.2 meters behind for a detected object classified asa “compact vehicle,” or 1 meter in front to 8 meters behind for adetected object classified as a “semi-truck.” Equivalently, operation524 may be accomplished by identifying a subset of LIDAR points that areassociated with distance measurements that lie outside the range.

This removal of the group of LIDAR points may be effective to removepoints attributable to an occluding object (e.g., occluding object 206)such as, for example, LIDAR points 400′. LIDAR points such as thosedepicted in 400′ may overwhelm the LIDAR points truly corresponding tothe detected object, such as LIDAR points 402′ and 404′, in some cases.This removal seeks to identify a secondary depth estimate.

At operation 526, the example process 500 may include sorting, bydistance, the subset of LIDAR points that are associated with depthmeasurements that lie outside the range, according to any of thetechniques discussed herein.

At operation 528, the example process 500 may include determining asecond weighted median of the sorted subset of LIDAR points andselecting, as a secondary depth estimate, a depth measurement associatedwith the second weighted median, according to any of the techniquesdiscussed herein. In other words, the first weighted median discussedabove is a weighted median of all the LIDAR points that are associatedwith the ROI, whereas the second weighted median is a weighted median ofa subset of those LIDAR points, e.g., those LIDAR points associated withdistances that lie outside the range discussed above and/or those LIDARpoints associated with distances that lie inside the range discussedabove.

At operation 530, the example process 500 may include comparingcharacteristics of the primary depth estimate and the secondary depthestimate and/or of LIDAR points associated therewith, according to anyof the techniques discussed herein. Operation 530 may differentiatebetween a true and false indication of depth for the detected object.For example, operation 530 may be conducted to distinguish between adepth estimate that corresponds to an occluding object instead of theobject itself. In some examples, operation 530 may include comparing anumber and/or a spatial density of LIDAR points associated with theprimary depth estimate to a number and/or a spatial density of LIDARpoints associated with the secondary depth estimate. For example, if thesubset of LIDAR points outside the range and therefore associated withthe secondary depth estimate is associated with a drastically lowernumber of LIDAR points than LIDAR points within the range, this may beindicative that the primary depth estimate is truly associated with thedetected object and that the LIDAR points associated with the primarydepth estimate are associated with a surface of the detected object. Insome examples, operation 536 may additionally or alternatively include adensity of the LIDAR points associated with the primary depth estimateand/or the secondary depth estimate, as a function of distance from acenter of the ROI. The denser and further away from the center of theROI that the secondary LIDAR points lie (i.e., LIDAR points outside therange), the more likely they may be associated with a second object(e.g., an occluding object) rather than the detected object.

Additionally, or alternatively, operation 530 may include determining,from the probability distribution generated by the monocular imagemodel, a probability and/or probability density associated with theprimary depth estimate and the secondary depth estimate. For example,the operation 530 may include determining that the primary depthestimate is associated with a probability and/or probability densitythat is lower than the secondary depth estimate. This may be more likelyto occur when the primary depth estimate is attributable to an occludingobject.

Additionally, or alternatively, operation 530 may include determine afirst fit of the primary depth estimate and a second fit of thesecondary depth estimate to an object track or a predicted object track.In some examples, the depth estimate that more closely corresponds tothe object track or the predicted object track may be chosen as thedepth estimate to be output.

In some examples, operation 530 may include a scoring function that isbased at least in part on any of the techniques discussed above and thatmay generate a score for the primary depth estimate and the secondarydepth estimate. In some examples, the object track technique may not beincluded in the scoring function, but may be used to break score ties.

At operation 532, the example process 500 may include outputting, basedat least in part on the comparison, the primary estimate, the secondarydepth estimate, and/or a mean or mode thereof to a vehicle controller(e.g., the vehicle planner), according to any of the techniquesdiscussed herein. For example, the depth estimate associated with ahigher score, higher probability and/or probability density, and/or moreclosely corresponding to an object track may be output to the vehicleplanner as the output depth estimate associated with the detected objectand relied on to control the autonomous vehicle. In some examples, theexample process 500 may include outputting a mean of the primary depthestimate and the secondary depth estimate if they lie within a thresholdvalue of each other or of a highest possible probability and/orprobability density.

At operation 534, the example process 500 may additionally includediscarding the depth estimate that was not output at operation 532, ifonly one depth estimate was selected for output, or associating thatdepth estimate with a second object (e.g., an occluding object),according to any of the techniques discussed herein. For example,operation 534 may include generating an indication that a second objectappears in the ROI. The perception engine 504 may use this indication tore-evaluate the original image and/or the ROI in an attempt to identifya second object. This may include varying object detection techniquesand/or machine-learning models used to do object detection. If a secondobject is detected, the other depth estimate that was not output inassociation with the first detected object, the other depth estimate maybe output to the vehicle planner in association with the second object.

Example Architecture

FIG. 6 is a block diagram of an example architecture 600 including anexample vehicle system 602 for controlling operation of at least onevehicle, such as an autonomous vehicle, using a depth estimate generatedaccording to any of the techniques discussed herein. In some examples,the vehicle system 602 may represent at least a portion of vehicle 108and/or 202. In some examples, this architecture may be used in othermachines to determine a depth of an object detected in an image.

In some examples, the vehicle system 602 may include processor(s) 604and/or memory 606. These elements are illustrated in combination in FIG.6, although it is understood that they may be separate elements of thevehicle system 602, and that components of the system may be implementedas hardware and/or software, in some examples.

Processor(s) 604 may include a uniprocessor system including oneprocessor, or a multiprocessor system including several processors(e.g., two, four, eight, or another suitable number). The processor(s)604 may be any suitable processor capable of executing instructions. Forexample, in various implementations, the processor(s) may begeneral-purpose or embedded processors implementing any of a variety ofinstruction set architectures (ISAs), such as the x86, PowerPC, SPARC,or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, eachprocessor 604 may commonly, but not necessarily, implement the same ISA.In some examples, the processor(s) 604 may include a central processingunit (CPU), a graphics processing unit (GPU), Field Programmable GateArrays (FPGA), Application Specific Integrated Circuit (ASIC), or acombination thereof.

The example vehicle system 602 may include memory 606. In some examples,the memory 606 may include a non-transitory computer readable mediaconfigured to store executable instructions/modules, data, and/or dataitems accessible by the processor(s) 604. In various implementations,the non-transitory computer readable media may be implemented using anysuitable memory technology, such as static random access memory (SRAM),synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or anyother type of memory. In the illustrated example, program instructionsand data implementing desired operations, such as those described above,are shown stored within the non-transitory computer readable memory. Inother implementations, program instructions, and/or data may bereceived, sent, or stored on different types of computer-accessiblemedia, such as non-transitory computer readable media, or on similarmedia separate from the non-transitory computer readable media. Anon-transitory, computer readable memory may include storage media ormemory media, such as flash memory (e.g., solid state memory), magneticor optical media (e.g., a disk) coupled to the example vehicle system602 via an input/output (“I/O”) interface 608. Program instructions anddata stored via a non-transitory computer readable medium may betransmitted by transmission media or signals such as electrical,electromagnetic, or digital signals, which may be conveyed via acommunication medium such as a network and/or a wireless link, such asmay be implemented via a network interface 610.

Furthermore, though illustrated as a single unit in FIG. 6, it isunderstood that the processor(s) 604 and memory 606 may be distributedamong multiple computing devices of the vehicle and/or among multiplevehicles, data centers, teleoperation centers, etc.

In some examples, the input/output (“I/O”) interface 608 may beconfigured to coordinate I/O traffic between the processor(s) 604, thememory 606, the network interface 610, sensor(s) 612, I/O devices 614,drive system 616, and/or any other hardware of the vehicle system 602.In some examples, the I/O devices 614 may include external and/orinternal speaker(s), display(s), passenger input device(s), etc. In someexamples, the I/O interface 608 may perform protocol, timing, or otherdata transformations to convert data signals from one component (e.g.,the non-transitory computer readable media) into a format suitable foruse by another component (e.g., processor(s)). In some examples, the I/Ointerface 608 may include support for devices attached through varioustypes of peripheral buses, such as the Peripheral Component Interconnect(PCI) bus standard, the Universal Serial Bus (USB) standard, or avariant thereof, for example. In some implementations, the function ofthe I/O interface 608 may be split into two or more separate components,such as a north bridge and a south bridge, for example. Also, in someexamples, some or all of the functionality of the I/O interface 608,such as an interface to the memory 606, may be incorporated directlyinto the processor(s) 604 and/or one or more other components of thevehicle system 602.

The example vehicle system 602 may include a network interface 610configured to establish a communication link (i.e., “network”) betweenthe vehicle system 602 and one or more other devices. For example, thenetwork interface 610 may be configured to allow data to be exchangedbetween the vehicle system 602 and another vehicle 618 via a firstnetwork 620, and/or between the vehicle system 602 and a remotecomputing system 622 via a second network 624. For example, the networkinterface 610 may enable wireless communication between another vehicle618 and/or the remote computing device 622. In various implementations,the network interface 610 may support communication via wireless generaldata networks, such as a Wi-Fi network, and/or telecommunicationsnetworks, such as, for example, cellular communication networks,satellite networks, and the like.

In some examples, the sensor data and/or perception data discussedherein may be received at a first vehicle and transmitted to a secondvehicle via the first network 620 and/or to a remote computing system622 via the second network 624.

The example vehicle system 602 may include sensor(s) 612, for example,configured to localize the vehicle system 602 in an environment, todetect one or more objects in the environment, to determine a depth ofan object detected in an image, to sense movement of the example vehiclesystem 602 through its environment, sense environmental data (e.g.,ambient temperature, pressure, and humidity), and/or sense conditions ofan interior of the example vehicle system 602 (e.g., passenger count,interior temperature, noise level). The sensor(s) 612 may include, forexample, one or more cameras 626 (e.g. RGB-cameras, intensity (greyscale) cameras, infrared cameras, UV cameras, depth cameras, stereocameras, monocular cameras), one or more LIDAR sensors 628, one or moreRADAR sensors 630, one or more magnetometers, one or more sonar sensors,one or more microphones for sensing sounds, one or more IMU sensors(e.g., including accelerometers and gyroscopes), one or more GPSsensors, one or more Geiger counter sensors, one or more wheel encoders(e.g., rotary encoders), one or more drive system sensors, a speedsensor, and/or other sensors related to the operation of the examplevehicle system 602.

In some examples, one or more of these types of sensors may bephase-locked (i.e., capturing data corresponding to substantially thesame portion of an environment of the vehicle at a substantially sametime) or asynchronous. For purposes of the techniques discussed herein,if the outputs of the camera(s) 626 and LIDAR(s) 628 and/or RADAR(s) 630are asynchronous, the techniques may include determining LIDAR dataand/or RADAR data that most closely corresponds to the camera data intime. For example, the perception engine 632 may make thisdetermination.

The example vehicle system 602 may include a perception engine 632, avision engine 634, that may include a monocular height machine-learning(ML) model 636, and a planner 638.

The vision engine 634 may include instructions stored on memory 606that, when executed by the processor(s) 604, cause the processor(s) 604to receive an image (e.g., a monocular image) of an environmentsurrounding the vehicle system 602; detect, in the image, an object inthe environment; generate an ROI (e.g., a bounding box, a pixel mask)that identifies a portion of the image as corresponding to the detectedobject; and/or generate, via the monocular height ML model 636 and basedat least in part on the ROI, an object classification received from theperception engine 632, and/or a height estimate of the detected object,a probability distribution. In some examples, the perception engine 632may generate the ROI and/or include the monocular height ML model 636and generate the probability distribution.

The monocular height ML model 636 may include the monocular image modeldiscussed in regard to FIG. 3 and/or FIG. 4, and/or as discussed in U.S.application Ser. No. 15/453,569, titled “Object Height Estimation fromMonocular Images” and filed Mar. 8, 2017. The monocular height ML model636 may include instructions stored on memory 606 that, when executed bythe processor(s) 604, cause the processor(s) 604 to receive an objectclassification, an image, and/or an ROI and generate a probabilitydistribution, according to a configuration of the layers of themonocular height ML model 636. In some examples, the probabilitydistribution may include probabilities indexed by distance, where anindividual distance is associated with an individual probability and/orprobability density that the individual distance is truly associatedwith the detected object. The vision engine 634 may transmit any of thedata determine and generated to the perception engine 632.

The perception engine 632 may include instructions stored on memory 606that, when executed by the processor(s) 604, cause the processor(s) 604to receive LIDAR data from a LIDAR device; determine LIDAR points thatcorrespond to a time that the image was captured and a region of theenvironment that corresponds to the ROI; generate scores for the LIDARpoints; and select, as a primary depth estimate, a weighted median ofthe LIDAR points, where the weighted median uses the scores as weights.The perception engine 632 may additionally or alternatively output theprimary depth estimate to the planner, determine a second depthestimate, and/or choose between the primary depth estimate and thesecondary depth estimate to send to the planner to associate with thedetected object and/or a second object, according to any of thetechniques discussed herein.

In some examples, the instructions may further configure theprocessor(s) 604 to receive sensor data from the sensor(s) 612 as input,and output data representative of, for example, one or more of the pose(e.g. position and orientation) of an object in the environmentsurrounding the example vehicle system 602, an object track associatedwith the object (e.g., a historic position, velocity, acceleration,and/or heading of the object over a period of time (e.g. 5 seconds)),and/or an object classification associated with the object (e.g. apedestrian, a passenger vehicle, a compact vehicle, a delivery truck, abicyclist). In some examples, perception engine 632 may be configured topredict an object trajectory of one or more objects. For example, theperception engine 632 may be configured to predict multiple objecttrajectories based on, for example, probabilistic determinations ormulti-modal distributions of predicted positions, trajectories, and/orvelocities associated with an object.

The perception engine 632 may transmit the output depth estimate, theROI, the image, an object classification associated with the detectedobject, an object track associated with the detected object, and/or anyother additional information that the planner 638 may use to generate atrajectory (e.g., object classifications, object tracks, vehicle pose).In some examples, the perception engine 632 and/or the planner 638 mayadditionally or alternatively transmit any of this data via the networkinterface 610 to the remote computing device 622 via network 624 and/oranother vehicle 618 via network 620, based at least in part on aconfidence generated by the monocular height ML model 636. In someexamples, the perception engine 632, the vision engine 634, and/or theplanner 638 may be located at another vehicle 618 and/or the remotecomputing device 622.

In some examples, the remote computing device 622 may include ateleoperations device. The teleoperations device may be a deviceconfigured to respond to an ROI, an output depth estimate, and/or a setof a primary depth estimate and a secondary depth estimate, where theperception engine 632 was unable to break a tie between the two, with anindication of whether output depth estimate is correct and/or aselection of the primary depth estimate and/or the secondary depthestimate as corresponding to the detected object and/or a second object.In additional or alternate examples, the teleoperations device maydisplay information related to the sensor data and/or the detectedobject, generated by the vision engine 634 and/or the perception engine632, that may be useful for receiving an input from a remote operator(“teleoperator”) corroborating or identifying a depth estimate. In suchexamples, the teleoperations device may include an interface forreceiving input, such as an indication that at least one of the depthestimates a true positive or a false positive, from the teleoperator. Insome examples, the teleoperations device may respond to the autonomousvehicle and/or additional autonomous vehicles corroborating theindication or identifying the indication as a false positive.

The planner 638 may include instructions stored on memory 606 that, whenexecuted by the processor(s) 604, cause the processor(s) 604 to generatedata representative of a trajectory of the example vehicle system 602,for example, using data representing a location of the example vehiclesystem 602 in its environment and other data, such as local pose data,and a position and/or track of the detected object, which may be basedon the output depth estimate and the ROI. In some examples, the planner638 may substantially continuously (e.g., every 1 or 2 milliseconds,though any receding horizon time is contemplated) generate a pluralityof potential trajectories with which to control the example vehiclesystem 602 and select one of the trajectories with which to control thevehicle. The selection may be based at least in part on a current route,depth estimate of the object, current vehicle trajectory, depth estimateof the object, and/or detected object trajectory data. Upon selecting atrajectory, the planner 638 may transmit the trajectory to the drivesystem 616 to control the example vehicle system 602 according to theselected trajectory.

In some examples, the perception engine 632, the vision engine 634, themonocular height ML model 636, and/or the planner 638 may furtherinclude specialized hardware such as, for example, a processor that issuited to running the perception engine (e.g., a graphics processor, anFPGA).

Example Clauses

A. A system comprising: one or more processors; and one or morecomputer-readable media storing instructions executable by the one ormore processors, wherein the instructions, when executed, cause thesystem to: receive an image of an environment from an image sensor;determine, based at least in part on the image, a region of interestthat identifies a portion of the image as representing an object in theenvironment; receive LIDAR points from a LIDAR device, the LIDAR pointsassociated with the region of interest and a time the image wascaptured; generate scores for the LIDAR points, wherein generating ascore for the LIDAR point comprises: determining, based at least in parton a probability distribution generated by a monocular image model, aprobability density associated with the depth measurement associatedwith the LIDAR point; and determining a factor, based at least in parton a distance in pixels between the LIDAR point projected into the imageand a center of the region of interest; and determine, using a weightedmedian calculation, a primary depth estimate of the object, whereinweights associated with the weighted median calculation comprise thescores.

B. The system as paragraph A recites, the instructions further cause thesystem to: select, as a subset of LIDAR points, LIDAR points that areassociated with depth measurements that are within a range of theprimary depth estimate; determine a second weighted median of the sortedLIDAR points; and determine, based at least in part on the secondweighted median, a secondary depth estimate of the object.

C. The system as either paragraph A or B recites, wherein the systemcomprises an autonomous vehicle, the camera and LIDAR being on theautonomous vehicle, and wherein the instructions further cause thesystem to: identify, based at least in part on the primary depthestimate or the secondary depth estimate, a position of the object inthe environment; and generate, based at least in part on the position ofthe object, a trajectory for controlling motion of the autonomousvehicle.

D. The system as any one of paragraphs A-C recites, wherein theinstructions further cause the system to: compare the primary depthestimate and the secondary depth estimate to an output of a monocularimage model; compare a first density of LIDAR points associated with theprimary depth estimate to a second density of LIDAR points associatedwith the secondary depth estimate; or compare the primary depth estimateand the secondary depth estimate to an object track associated with theobject.

E. The system as any one of paragraphs A-D, wherein generating the scorefor the LIDAR point comprises multiplying the probability density by thefactor.

F. A computer-implemented method of determining a distance from an imageplane to an object, the method comprising: receiving LIDAR data andimage data of an environment; determining a region of interestassociated with the object detected in the environment; determiningLIDAR points of the LIDAR data that correspond to the region ofinterest; generating scores for the LIDAR points, wherein generating ascore for a LIDAR point comprises: determining a factor based at leastin part on a distance from a center of the region of interest to aprojection of the LIDAR point onto the image; determining a probabilitydensity of a depth measurement associated with the LIDAR point; andgenerating the score based at least in part on the probability densityand the factor; determining, based at least in part on the scores, aweighted median of the LIDAR points; and identifying, as a primary depthestimate, a depth measurement associated with the weighted median.

G. The computer-implemented method as paragraph F recites, whereindetermining the factor comprises evaluating a Gaussian centered at thecenter of the region of interest using the projection of the LIDAR pointonto the image.

H. The computer-implemented method as either paragraph F or G recites,wherein determining the probability density comprises generating aprobability distribution over a range of depths, via a machine-learningmodel and based at least in part on a classification of the object.

I. The computer-implemented method as any one of paragraphs F-H recites,wherein generating the score comprises multiplying the probabilitydensity by the factor.

J. The computer-implemented method of paragraph F, further comprising:identifying a subset of LIDAR points associated with distances that meetor exceed a range of depth values that comprises the primary depthestimate; sorting the subset of LIDAR points by distances associatedwith the subset of LIDAR points; determining, based at least in part onscores associated with the subset and the sorting, a second weightedmedian; and identifying, as a secondary depth estimate, a depthmeasurement associated with the second weighted median.

K. The computer-implemented method as any one of paragraphs F-J recites,wherein the range of depth values varies from a point 0.H meters lessthan the primary depth estimate to A.F more than the primary depthestimate.

L. The computer-implemented method as any one of paragraphs F-K recites,further comprising: choosing, as an output depth, the primary depthestimate or the secondary depth estimate based at least in part on atleast one of: comparing a first probability density or a firstprobability associated with the first depth estimate by evaluating theprobability distribution using the first depth estimate, to a secondprobability density or a second probability associated with the seconddepth estimate by evaluating the probability distribution using thesecond depth estimate; comparing a first density of LIDAR pointsassociated with the primary depth estimate to a second density of LIDARpoints associated with the secondary depth; or comparing the primarydepth estimate and the secondary depth estimate to an object trackassociated with the object.

M. The computer-implemented method as any one of paragraphs F-L recites,wherein choosing the secondary depth estimate further comprises:indicating an existence of an occluding object that occludes at leastpart of the object; and associating the primary depth estimate with theoccluding object and the secondary depth estimate with the object.

N. The computer-implemented method as any one of paragraphs F-M recites,further comprising: sending the output depth to a controller of anautonomous vehicle; and generating, based at least in part on the outputdepth, a trajectory, the trajectory configured to cause the autonomousvehicle to traverse a portion of the environment.

O. A non-transitory computer-readable medium having a set ofinstructions that, when executed, cause one or more processors toperform operations comprising: receiving, from a camera, an image of anenvironment that comprises an object; receiving a region of interestrepresenting a location of the object in the image; receiving, from apoint cloud sensor, point cloud data; determining, from the point clouddata, point cloud points that correspond with the region of interest;determining, based at least in part on the image, a probabilitydistribution of depths; generating, based at least in part on relativecoordinates of the point cloud points in an image space associated withthe image and based at least in part on a position of the point cloudpoints relative to depths specified by the probability distribution,scores for the point cloud points; determining, by a weighted mediancalculation, a weighted median based at least in part on the scores; andidentifying a depth measurement associated with the weighted median as afirst depth estimate of the object.

P. The non-transitory computer-readable medium as paragraph O recites,the operations further comprising: determining a subset of point cloudpoints associated with depth measurements that lie outside a range ofdepths from the first depth estimate; determine a second weighted medianof the subset of point cloud points; and identify a second distanceassociated with the second weighted median as a second depth estimate ofthe object.

Q. The non-transitory computer-readable medium as either paragraph O orP recites, the operations further comprising: comparing at least one of:a first probability density or a first probability associated with thefirst depth estimate by evaluating the probability distribution usingthe first depth estimate, to a second probability density or a secondprobability associated with the second depth estimate by evaluating theprobability distribution using the second depth estimate; a firstdensity of point cloud points associated with the first depth estimateto a second density of point cloud points associated with the seconddepth estimate; or the first depth estimate and the second depthestimate to an object track associated with the object; and associating,based at least in part on the comparing, one of the first depth estimateor the second depth estimate with the object.

R. The non-transitory computer-readable medium as any one of paragraphsO-Q recites, the operations further comprising generating a trajectoryfor controlling motion of an autonomous vehicle based at least in parton at least one of the first depth estimate or the second depthestimate.

S. The non-transitory computer-readable medium as paragraph O recites,wherein determining the factor is based at least in part on evaluating aGaussian distribution centered at the center of the region of interestwith respect to a distance of the projection of the LIDAR point from thecenter of the region of interest.

T. The non-transitory computer-readable medium as any one of paragraphsO-S recites, wherein generating the score for the LIDAR point comprisesmultiplying the probability density by the factor.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described. Rather,the specific features and acts are disclosed as example forms ofimplementing the claims.

The modules described herein represent instructions that can be storedin any type of computer-readable medium and can be implemented insoftware and/or hardware. All of the methods and processes describedabove can be embodied in, and fully automated via, software code modulesand/or computer-executable instructions executed by one or morecomputers or processors, hardware, or some combination thereof. Some orall of the methods can alternatively be embodied in specialized computerhardware.

Conditional language such as, among others, “can,” “could,” “may” or“might,” unless specifically stated otherwise, are understood within thecontext to present that certain examples include, while other examplesdo not include, certain features, elements and/or steps. Thus, suchconditional language is not generally intended to imply that certainfeatures, elements and/or steps are in any way required for one or moreexamples or that one or more examples necessarily include logic fordeciding, with or without user input or prompting, whether certainfeatures, elements and/or steps are included or are to be performed inany particular example.

Conjunctive language such as the phrase “at least one of X, Y or Z,”unless specifically stated otherwise, is to be understood to presentthat an item, term, etc. can be either X, Y, or Z, or any combinationthereof, including multiples of each element. Unless explicitlydescribed as singular, “a” means singular and plural.

Any routine descriptions, elements or blocks in the flow diagramsdescribed herein and/or depicted in the attached figures should beunderstood as potentially representing modules, segments, or portions ofcode that include one or more computer-executable instructions forimplementing specific logical functions or elements in the routine.Alternate implementations are included within the scope of the examplesdescribed herein in which elements or functions can be deleted, orexecuted out of order from that shown or discussed, includingsubstantially synchronously, in reverse order, with additionaloperations, or omitting operations, depending on the functionalityinvolved as would be understood by those skilled in the art.

It should be emphasized that many variations and modifications can bemade to the above-described examples, the elements of which are to beunderstood as being among other acceptable examples. All suchmodifications and variations are intended to be included herein withinthe scope of this disclosure and protected by the following claims.

What is claimed is:
 1. A system comprising: one or more processors; andone or more computer-readable media storing instructions executable bythe one or more processors, wherein the instructions, when executed,cause the system to: receive an image of an environment from an imagesensor; determine, based at least in part on the image, a region ofinterest that identifies a portion of the image as representing anobject in the environment; receive LIDAR points from a LIDAR device, theLIDAR points associated with the region of interest and a time the imagewas captured; generate scores for the LIDAR points, wherein generating ascore for the LIDAR point comprises: determining, based at least in parton a probability distribution generated by a monocular image model, aprobability density associated with the depth measurement associatedwith the LIDAR point; and determining a factor, based at least in parton a distance in pixels between the LIDAR point projected into the imageand a center of the region of interest; and determine, using a weightedmedian calculation, a primary depth estimate of the object, whereinweights associated with the weighted median calculation comprise thescores.
 2. The system as claim 1 recites, the instructions further causethe system to: select, as a subset of LIDAR points, LIDAR points thatare associated with depth measurements that are within a range of theprimary depth estimate; determine a second weighted median of the sortedLIDAR points; and determine, based at least in part on the secondweighted median, a secondary depth estimate of the object.
 3. The systemas claim 2 recites, wherein the system comprises an autonomous vehicle,the camera and LIDAR being on the autonomous vehicle, and wherein theinstructions further cause the system to: identify, based at least inpart on the primary depth estimate or the secondary depth estimate, aposition of the object in the environment; and generate, based at leastin part on the position of the object, a trajectory for controllingmotion of the autonomous vehicle.
 4. The system as claim 2 recites,wherein the instructions further cause the system to: compare theprimary depth estimate and the secondary depth estimate to an output ofa monocular image model; compare a first density of LIDAR pointsassociated with the primary depth estimate to a second density of LIDARpoints associated with the secondary depth estimate; or compare theprimary depth estimate and the secondary depth estimate to an objecttrack associated with the object.
 5. The system as claim 1 recites,wherein generating the score for the LIDAR point comprises multiplyingthe probability density by the factor.
 6. A computer-implemented methodof determining a distance from an image plane to an object, the methodcomprising: receiving LIDAR data and image data of an environment;determining a region of interest associated with the object detected inthe environment; determining LIDAR points of the LIDAR data thatcorrespond to the region of interest; generating scores for the LIDARpoints, wherein generating a score for a LIDAR point comprises:determining a factor based at least in part on a distance from a centerof the region of interest to a projection of the LIDAR point onto theimage; determining a probability density of a depth measurementassociated with the LIDAR point; and generating the score based at leastin part on the probability density and the factor; determining, based atleast in part on the scores, a weighted median of the LIDAR points; andidentifying, as a primary depth estimate, a depth measurement associatedwith the weighted median.
 7. The computer-implemented method as claim 6recites, wherein determining the factor comprises evaluating a Gaussiancentered at the center of the region of interest using the projection ofthe LIDAR point onto the image.
 8. The computer-implemented method asclaim 6 recites, wherein determining the probability density comprisesgenerating a probability distribution over a range of depths, via amachine-learning model and based at least in part on a classification ofthe object.
 9. The computer-implemented method of claim 6, whereingenerating the score comprises multiplying the probability density bythe factor.
 10. The computer-implemented method as claim 6 recites,further comprising: identifying a subset of LIDAR points associated withdistances that meet or exceed a range of depth values that comprises theprimary depth estimate; sorting the subset of LIDAR points by distancesassociated with the subset of LIDAR points; determining, based at leastin part on scores associated with the subset and the sorting, a secondweighted median; and identifying, as a secondary depth estimate, a depthmeasurement associated with the second weighted median.
 11. Thecomputer-implemented method as claim 10 recites, wherein the range ofdepth values varies from a point 0.8 meters less than the primary depthestimate to 1.6 meters more than the primary depth estimate.
 12. Thecomputer-implemented method as claim 10 recites, further comprising:choosing, as an output depth, the primary depth estimate or thesecondary depth estimate based at least in part on at least one of:comparing a first probability density or a first probability associatedwith the first depth estimate by evaluating the probability distributionusing the first depth estimate, to a second probability density or asecond probability associated with the second depth estimate byevaluating the probability distribution using the second depth estimate;comparing a first density of LIDAR points associated with the primarydepth estimate to a second density of LIDAR points associated with thesecondary depth; or comparing the primary depth estimate and thesecondary depth estimate to an object track associated with the object.13. The computer-implemented method as claim 12 recites, whereinchoosing the secondary depth estimate further comprises: indicating anexistence of an occluding object that occludes at least part of theobject; and associating the primary depth estimate with the occludingobject and the secondary depth estimate with the object.
 14. Thecomputer-implemented method as claim 12 recites, further comprising:sending the output depth to a controller of an autonomous vehicle; andgenerating, based at least in part on the output depth, a trajectory,the trajectory configured to cause the autonomous vehicle to traverse aportion of the environment.
 15. A non-transitory computer-readablemedium having a set of instructions that, when executed, cause one ormore processors to perform operations comprising: receiving, from acamera, an image of an environment that comprises an object; receiving aregion of interest representing a location of the object in the image;receiving, from a point cloud sensor, point cloud data; determining,from the point cloud data, point cloud points that correspond with theregion of interest; determining, based at least in part on the image, aprobability distribution of depths; generating, based at least in parton relative coordinates of the point cloud points in an image spaceassociated with the image and based at least in part on a position ofthe point cloud points relative to depths specified by the probabilitydistribution, scores for the point cloud points; determining, by aweighted median calculation, a weighted median based at least in part onthe scores; and identifying a depth measurement associated with theweighted median as a first depth estimate of the object.
 16. Thenon-transitory computer-readable medium as claim 15 recites, theoperations further comprising: determining a subset of point cloudpoints associated with depth measurements that lie outside a range ofdepths from the first depth estimate; determine a second weighted medianof the subset of point cloud points; and identify a second distanceassociated with the second weighted median as a second depth estimate ofthe object.
 17. The non-transitory computer-readable medium as claim 16recites, the operations further comprising: comparing at least one of: afirst probability density or a first probability associated with thefirst depth estimate by evaluating the probability distribution usingthe first depth estimate, to a second probability density or a secondprobability associated with the second depth estimate by evaluating theprobability distribution using the second depth estimate; a firstdensity of point cloud points associated with the first depth estimateto a second density of point cloud points associated with the seconddepth estimate; or the first depth estimate and the second depthestimate to an object track associated with the object; and associating,based at least in part on the comparing, one of the first depth estimateor the second depth estimate with the object.
 18. The non-transitorycomputer-readable medium as claim 16 recites, the operations furthercomprising generating a trajectory for controlling motion of anautonomous vehicle based at least in part on at least one of the firstdepth estimate or the second depth estimate.
 19. The non-transitorycomputer-readable medium as claim 15 recites, wherein determining thefactor is based at least in part on evaluating a Gaussian distributioncentered at the center of the region of interest with respect to adistance of the projection of the LIDAR point from the center of theregion of interest.
 20. The non-transitory computer-readable medium asclaim 15 recites, wherein generating the score for the LIDAR pointcomprises multiplying the probability density by the factor.